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INTRODDCTIOH 

Public concern in Ontario over the possible toxic effects of 
exposure to environmental contaminants is high in the wake of 
past highly publicized events elsewhere, for example, Love Canal, 
Three Mile Island, and Chernobyl. Local situations, such as the 
Upper Ottawa Street Landfill site in Hamilton, reinforce concerns 
and give them more immediacy. Recent disclosures regarding PCB 
disposal sites, radon gas emissions and electro-magnetic 
radiation also contribute to uncertainty and anxiety over health 
risks. Equally, the prospect of the future location of new solid 
and liquid waste disposal sites compounds public fears and 
generates community reaction. 

Research on the human health consequences of environmental 
contaminants has focussed primarily on possible physical effects 
(e.g. cancer outcomes). Increasing attention is now being given 
to psychosocial morbidity, in part as a result of research 
undertaken in the wake of such high profile events as the 
accident at Three Mile Island (e.g. Sorenson et al . , 1987; Vvner 
1988) . ' 

Superficially, the link between exposure to contaminants and 
negative psychosocial outcomes (e.g. anxiety, depression 
domestic and interpersonal problems) is self-evident. On closer 
examination, however, current knowledge based on past research is 
fragmentary, and a necessary starting point for future 
investigation is a critical appraisal of the existing literature 
Recent reviews (Madisso, 1985; Michalenko and Lerner, 1987) 
provide a general description of the types of research attempted 
and the main findings reported but do not provide a critical 
appraisal of the evidence contained in previous studies. 

This paper describes the development and application of a 
methodology for assessing the literature on the psychosocial 
impacts of environmental contaminants. The methodology was 
devised as part of a feasibility study for which the eventual end 
point is a protocol for research on the psychosocial impacts of 
environmental contaminants in Ontario. Although the intended 
purpose of the methodology is quite specific, the governing 
principles of critical appraisal on which it is based are quite 
general. It follows that the method has potential application to 
the assessment of research on a broad range of environmental 
health issues. 



OVERVIEW OP THE METHODOLOGY 

The methodology was designed with 
appraisal objectives in view: 



the following critical 



To assess the strength of the evidence for causal links 
between exposure to environmental contaminants and 
psychosocial impacts. 



To develop a conceptual model of the process linking 
exposure to environmental contaminants and psychosocial 
impacts . 

To assess the strengths and weaknesses of feasible 
research designs for the study of psychosocial impacts. 

To evaluate existing instruments for measuring 
psychosocial impacts of exposure to environmental 
contaminants. 

To assess analytical/statistical methods for estimating 
relationships between exposures and outcomes. 

To evaluate approaches for determining the exposure 
characteristics of populations. 

To identify population sub-groups particularly 
susceptible to psychosocial impacts. 

To review the implications for public education of the 
psychosocial impacts of exposure to environmental 
contaminants. 

The methodology divides into three stages: literature search, 
screening and appraisal. The first two are straightforward and 
require only brief comment. The appraisal procedure is more 
complex and needs fuller description and explanation. 

Literature Search: 

Increasingly, researchers have come to depend on computer-based 
bibliographic search methods. The scope has increased as the size 
and number of computerized bibliographic data bases has 
multiplied. In the present case ten different data bases were 
searched (BIOSIS, Enviroline, Environmental Bibliography, 
Medline, NTIS, Pollution Abstracts, Popline, Psyc Info, 
Sociological Abstracts, and Toxline) . 

The initial search strategy was to use combinations of generic 
keywords for both exposure (e.g. 'environment', 'toxic', 
'pollution', 'waste', 'risk') and impacts (e.g. ' (di)stress' , 
'psycho', 'socio', 'health', 'behaviour'). This global approach 
was ineffective and inefficient. It generated literally thousands 
of citations, many of which were irrelevant, and several key 
citations were missed. 

A revised strategy involved searching each data base for "key" 
citations to determine salient keywords specific to that 
bibliography. These were then used for a second comprehensive 
search. This was far more productive, but still failed to 
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identify some important references uncovered by preliminary 
manual search methods. ' 

This experience is not unusual for users of bibliographic data 
bases in situations where the target literature is quite diverse 
(i.e. in this case, a wide range of relevant environmental 



_ange of relevant environmental 
contaminants and psychosocial outcomes) and therefore spread 
across a wide range of journals and classified using an 
unstandardized set of keywords. Consultation with other 
researchers in the field confirmed similar difficulties in 
identifying relevant literature on this topic using computerized 
search methods . 

The net result was that it was necessary to supplement the 
automated search procedures with conventional manual methods. 
This involved a systematic search of all issues of relevant 
journals back to 1980. The journal list was defined in two ways: 
first any journal containing a previously identified relevant 
?f^»H e , W "K 1 K Cl V ded ''.- Se , cond ' P°tentially relevant journals 
i« % high im P act " in the Science Citations Index were 

added. 

Citations identified by this combination of automated and manual 
methods were entered into a bibliographic data base using Word 
Perfect 4.2 for efficient data storage and retrieval. 

Literature Screening 

The list of citations generated by the search inevitably included 
items of varying relevance for the study objectives. A screeninq 
form was developed to identify articles for full appraisal. The 
screening form included three classification criteria: type of 
effect (physical, psychosocial), type of exposure (environmental, 
occupational) and type of publication (original research 
methodology, review, conceptual, book, discussion/commentary)! 
All citations dealing with relationships between environmental 
exposures and psychosocial effects qualified for full appraisal. 

LITERATURE ASSESSMENT 

The literature on the psychosocial impacts of environmental 
contaminants is diverse and calls for an innovative appraisal 
methodology. A literature assessment form was developed to 
fulfill two purposes: to summarize the content of each paper; and 
to evaluate the strength of evidence with reference to 
standardized methodological criteria. The form was the product of 
extensive discussion among the research team. Draft versions were 
pretested and revised as appropriate. A user's manual was written 
to ensure consistency in application of the form by assessors. 
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Content Summary Criteria 

Check lists were developed to summarize the content of articles 
under five headings: study design; health outcome; exposure 
classification; sample design; and analysis methods. 

Study design is a very important consideration in assessing the 
strength of evidence. The epidemiologic literature provides 
detailed discussion of design architecture and its implications 
for causal inference. Designs for environmental epidemiology are 
typically one of three generic types: cohort, case-control and 
cross-sectional (Marsh and Caplan, 1986; Kasl, 1985). In 
addition, a variety of other designs are used (e.g. ecologic, 
time series and ethnographic) . 

A "diagnostic tree" was devised to determine the design used in 
each of the studies under review (Fig. 1) . This consisted of a 
sequence of diagnostic questions. 'Yes'/'no' answers to each 
question defined a specific design type. For example, a design 
was classified as 'epidemiologic'/'individual level '/'cross- 
sectional' if the answers to the diagnostic questions were as 
follows: 

"Does the study review and quantitatively 
summarize 'primary data' studies? NO 

"Does the study attempt quantitative 
inferences about the risk of any health 
outcome associated with exposure to 
environmental hazards?" YES 

"Does the study analyze a risk/exposure 
relationship using human individuals as the 
unit of analysis?" YES 

"Are all exposures and health outcomes 
referent to the same point in time?" YES 

The classification of health outcomes in the target literature is 
complex because of the wide range of physical and psychosocial 
effects reported. Although not the primary focus of the inquiry, 
it was essential to include physical effects because they are 
frequently reported in combination with psychosocial outcomes. 
Indeed, self-reported physical symptoms may in themselves 
represent psychosocial effects. A comprehensive checklist was 
devised to reflect current thinking about morbidity. Lists were 
written under five major categories: disease outcome (by body 
system and type); objectively measured dysfunction (physical, 
neuropsychological and psychological) , illness stigmata 
(physical, emotional and cognitive complaints), illness behaviour 
(health concerns and help-seeking), and disability 
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(vocational/educational, activities of daily living, domestic and 
interpersonal, social and recreational). 

Exposure definition is also complex. It is important to consider 
several aspects. Six classification categories were included in 
the appraisal form: experience, source, pathway, culprit, 
economic benefit and public familiarity. Several of these deserve 
brief comment. 

It is important to document the nature of the exposure 
experience. The distinction is made between actual (past or 
present), future and perceived exposure, although these are not 
mutually exclusive. Psychosocial effects are frequently reported 
in situations where actual exposure either has never occurred or 
is in doubt. This is obviously the case where effects occur in 
advance of the presence of a contaminant (e.g. anxiety in 
anticipation of a future waste disposal site) . Equally, effects 
are reported in response to perceived past or present exposure 
which may bear little relation to measures of actual 
contamination. Another important distinction is between acute 
(e.g. accident situations) and chronic (e.g. industrial 
emissions) exposure. The (supposed) source of contamination (e.g. 
landfill site, heavy industry, ambient conditions) is significant 
given the perceived risk attached to certain types of facility 
(e.g. nuclear power plants) and the heightened awareness and 
mistrust which may ensue. The presence or absence of a 'culprit' 
for contamination affects whether there is an identifiable target 
for the expression of public concern and may influence the types 
of coping responses in an impacted community. Risk perception and 
reported effects may be related to the economic benefit, if any, 
attached to the source of contamination. This is especially 
evident in single industry towns where the pollution source is 
the major employer and fundamental to the economic base of the 
community. The inclusion of familiarity reflects the importance 
of prior public knowledge of contaminants and their possible 
health consequences. Knowledge about local risks may arise in a 
variety of ways; for example, from media coverage of comparable 
situations elsewhere or from the activities of local 
environmental groups. 

For sample design and analysis, the appraisal form included items 

to cover details of sample size, composition and 

representativeness, and ' the specific types of statistical or 
other analytical methods employed. 

Assessment of Evidence Criteria 

The second part of the appraisal form was designed to provide a 
systematic basis for evaluating the strength of evidence reported 
in each of the papers under review. While there exist general 
epidemiologic principles for an evaluation exercise of this sort 
(Frank et al., 1988; McMaster Univ., 1981), there was no "off the 
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shelf" instrument suited to the specific literature in view 
Items were developed for inclusion in the appraisal form to 
to ,111 *?" f ° llow £"9 »ajor !•■«»■: the measurement of exposure 
to contaminants; the measurement of health outcomes; control of 
extraneous/confounding variables; strength of association- 
evaluation of study results; global assessment of credibility 

It follows from the earlier discussion on the nature of the 
TXSS? experience related to psychosocial effects that the 
identification and measurement of exposure is inherently 
S^if ; Tnorou 9 h documentatation of the type, intensity and 
duration of exposure is unlikely in most studies. Typically 
effects are reported in response to an uncertain 'dose' of 
uncertain contaminants. The problem is compounded when perceived 
exposure is considered, given that this can occur in the absence 
withst»n^ C nl Ua X contamination at all. These complications not 
withstanding, items were included in the appraisal form to 
~«£ 2 S T° W V„ if at a11 ' exposure «g^ts had been identified and 
measured. In the same context, items to determine the use and 
f^T 3 ^ * control groups were developed. An item was also 
included to assess exposure to 'health alarms' in the community 
^; g ; f 3< , hea " h warnings, media reports, etc.) recognizing 
reposed eflects 1116 " 08 ""* COUld aX * rt °" PUbliC " act?on ™« 

The assessment of outcomes was considered in terms. of 
appropriateness and accuracy. Appropriateness is related to the 
biological/psychological and epidemiological plausibility of the 
outcomes reported with reference to the contaminants in question? 
This is typically a difficult assessment to make for psychosocial 
outcomes because of their complex and uncertain causation and the 
absence of specific links to particular exposures. In short, a 
^nL rai y e ou tcomes are plausibly related to an equally wide 
range of exposures. An additional issue is the timing of the 

Mr?2T%h™* latl }'\* t0 v t "'.* Xpo ' ur * taking ^oount of any latency 
period that might be involved. The assessment of accuracy 

VlV fLM ltIVi S. ,U ' feW false negatives), specificity 
(i.e. few false positives) and bias (e.g. self-reported effects). 
^iinT ,™ S V? f self-report measures of undetermined 
reliability and validity complicates the assessment of accuracy. 
Confounding variables are factors related to both exposure and 
outcome which if not controlled in the study design or analysis? 
rt! n™ Z, falSe inference about an exposure-outcome 

relationship. From the standpoint of assessment, it is relevant 
„° "ftermine the use and characteristics of control groups and 
iDeciScanv t Jf Ir i co fP ara biHty to the exposed population, 
specifically it is important to determine the adequacy of 
matching and/or the methods used to adjust for possible 
uncfrtaf" ln -P" anal * sis - "ere again, the compflx and 
uncertain causation of psychosocial effects is problematic 
because it is difficult to compile a full list of potential 
confounders. it is reasonable to suppose that P various 
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socioderaographic variables (age, gender, socioeconomic status) 
are relevant. It nay be important to consider psychological 
factors as well (e.g. personality traits) . 

Strength of association corresponds to the change in risk of an 
outcome occurring after exposure to an imputed cause. Two 
questions arise in assessing relative risk, namely its clinical 
or practical significance and its statistical significance. Items 
were included in the appraisal form to assess both. These two 
aspects of significance are not necessarily coincident. A 
relative risk may be judged clinically significant though not 
statistically significant (e.g. small sample sizes may preclude 
statistical significance despite a relative risk of clinically 
significant proportions) . Alternatively, a statistically 
significant result may have uncertain clinical significance (e.g. 
if the outcome in question has undetermined health consequences) . 
A third consideration regarding strength of association is the 
demonstration of a dose-response relationship (i.e. risk 
increases monotonically with increased exposure) and an item was 
included to judge this, where a relationship is present there is 
strong evidence of a true effect. On the other hand, its absence 
may be due to many factors and it is therefore not a definitive 
criterion. The frequent absence of precise exposure measures for 
reasons already described precludes the calculation of dose- 
response relationships in many studies. 

The overall evaluation of reported results comes down to the 
question of whether the demonstrated exposure-effect link is true 
positive, false positive, true negative or false negative. All of 
the evaluation criteria previously described have to be taken 
into account to make this judgment. Three items were included in 
the appraisal form to judge the possibility of a false positve 
association, a false negative association and the consistency 
with other studies of the same exposure-effect relationship. 
The last of these is made more difficult if the exposure and 
outcome in question are not well understood from previous 
research. 

Three final items were included on the appraisal form to provide 
a global assessment based on all of the previous criteria. The 
first required a judgment as to whether the evidence supported an 
association for each exposure-outcome relationship examined. The 
second item determined if there were other factors which might 
cast doubt on the validity of the association (e.g. temporality, 
biological/psychological credibility, consistency with other 
studies) . The last item called for a bottom line judgment on the 
credibility of the study. 
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APPLICATION OP THE METHODOLOGY 

Results of the Literature Search and Screening 

The combination of automated and manual literature search methods 
previously described resulted in the compilation of a 
bibliography containing 296 citations. These were classified 
according to the three major criteria (type of effect, type of 
exposure, and type of publication) on the screening form, 
classification codes were included with each citation in the 
bibliography which was stored as a Word Perfect (secondary merge) 
file. Search and selection within this file allowed for the 
generation of subfiles defined by user-specified requirements 
(e.g. original research studies of relationships between 
environmental exposures and psychosocial effects.) 

The frequency distribution of citations according to the 
screening criteria is summarized in Table 1. As intended by the 
choice of search strategy and keyword selection, most citations 
deal with psychosocial effects (263) and environmental exposures 
(192). 164 studies involve a combination of these two. Both 
physical and psychosocial effects are quite frequently examined 
within a single study (68 citations) . An exposure-effect 
relationship was examined in 175 cases. In terms of type of 
publication, 'original research' is the most frequent category 
(160) followed by 'conceptual' (81) and 'discussion/commentary' 
(64). 

The subset of studies for full appraisal was defined as those 
which examined a relationship between environmental exposure and 
psychosocial effects based on original research. 52 citations 
satisfied these criteria. These are the key papers for addressing 
the first of the objectives previously listed: to assess the 
strength of evidence for causal links between exposure to 
environmental contaminants and psychosocial impacts. The balance 
of the citations are the basis for a more general review of the 
literature in fulfillment of the other stated objectives. 

Appraisal Procedure 

Prior to the appraisal of the 52 papers, an inter-rater 
reliability test was conducted. This involved three team members 
making independent assessments of six papers using the appraisal 
form. The papers were selected to represent the range of topics 
in the full set of 52. The results showed a high degree of 
consistency among raters. The test did reveal some differences in 
interpretation of specific items on the form. These were reported 
to the full team and agreement was reached on standard 
interpretations . 

At the time of writing the appraisal is nearing completion. The 
procedure has been for two team members to make independent 
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assessments of each paper. These are then checked for 
consistency. Differences in judgments on content summary items 
are in most instances resolved by checking the relevant factual 
details in the paper. Differences on the assessment of evidence 
items are resolved by discussion between the two asessors. Where 
this fails to achieve consensus, a third party from the research 
team is called on to reach a majority judgment. Consistent with 
the pretest findings, the differences have been relatively few 
and resolution has in almost all cases been achieved by 
consultation between the two original assessors. 

Customized software has been developed for data entry from the 
appraisal form. This provides a VDT screen image of the form to 
facilitate data entry. The package is menu driven with facilities 
to modify and retrieve data and to copy user defined files. The 
modification procedure allows for the addition, deletion or 
correction of information. The retrieval facility is flexible 
allowing access by title, author, key words or particular 
appraisal items (e.g. study design, outcome, exposure) . The 
software can create files for input to other programs (e.g. 
SPSSX) to facilitate further analysis (e.g. contingency 
analyses) . 
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TABLE 1 
FREQUENCY DISTRIBUTION OF CITATIONS 

CATEGORY FREQUENCY 

HEALTH EFFECT 

Psychosocial 263 
Physical 79 

EXPOSURE 

Environmental 192 
Occupational 82 

EXPOSURE-EFFECT RELATIONSHIP ASSESSED 175 

TYPE OF STUDY 

Original Research 160 
Conceptual 81 

Methodology 63 

Review 53 

Commentary 64 

Book 22 

TOTAL CITATIONS 296 
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